Robustness Versus Fidelity In Natural Language Understanding

A number of issues arise when trying to scaleup natural language understanding (NLU) tools designed for relatively simple domains (e.g., flight information) to domains such as medical advising or tutoring where deep understanding of user utterances is necessary. Because the subject matter is richer, the range of vocabulary and grammatical structures is larger meaning NLU tools are more likely to encounter out-of-vocabulary words or extra-grammatical utterances. This is especially true in medical advising and tutoring where users may not know the correct vocabulary and use common sense terms or descriptions instead. Techniques designed to improve robustness (e.g., skipping unknown words, relaxing grammatical constraints, mapping unknown words to known words) are effective at increasing the number of utterances for which an NLU subsystem can produce a semantic interpretation. However, such techniques introduce additional ambiguity and can lead to a loss of fidelity (i.e., a mismatch between the semantic interpretation and what the language producer meant). To control this trade-off, we propose semantic interpretation confidence scores akin to speech recognition confidence scores, and describe our initial attempt to compute such a score in a modularized NLU sub-system.